Embedded OTel Collector Config #1314

aphralG · 2025-10-01T16:43:29Z

Proposed changes

This change allows a user to list paths to their own Otel configs, these are then merged with the default Agent collector config. The last config specified takes priority over the rest if they have the same values declared.

collector:
  additional_config_paths:
    - "/configs/my_config.yaml"
    - /etc/nginx-agent/nginx-agent-otelcol.yaml

To make debugging easier the merged OTel config which OTel is running with is outputted to /var/lib/nginx-agent/opentelemetry-collector-agent-debug.yaml

To make modifying the NGINX Receiver and NGINX Plus Receiver easier if there is only one instance running thee instance ID is no longer needed be needed for the name. This will make modifying the collection_interval setting for example using your own OTel config easier as you no longer need the instanceID

  nginx/123456789:
    instance_id: "123"
    api_details:
      url: "http://localhost:80/status"
      listen: ""
      location: ""
      ca: ""
    collection_interval: 30s
    access_logs:
      - log_format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\"\"$upstream_cache_status\""
        file_path: "/var/log/nginx/access-custom.conf"

will now be

  nginx:
    instance_id: "123456789"
    api_details:
      url: "http://localhost:80/status"
      listen: ""
      location: ""
      ca: ""
    collection_interval: 30s
    access_logs:
      - log_format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\"\"$upstream_cache_status\""
        file_path: "/var/log/nginx/access-custom.conf"

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING document
I have run make install-tools and have attached any dependency changes to this pull request
If applicable, I have added tests that prove my fix is effective or that my feature works
If applicable, I have checked that any relevant tests pass after adding my changes
If applicable, I have updated any relevant documentation (README.md)
If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

spencerugbo · 2025-10-06T13:40:17Z

internal/config/testdata/nginx-agent.conf

  config_path: "/etc/nginx-agent/nginx-agent-otelcol.yaml"
+  additional_config_paths:
+    - "/configs/my_config.yaml"
+    - /etc/nginx-agent/nginx-agent-otelcol.yaml


Should this be commented?

Suggested change

- /etc/nginx-agent/nginx-agent-otelcol.yaml

- "/etc/nginx-agent/nginx-agent-otelcol.yaml"

It's without the quotes because I wanted to check it would work with and without the quotes

dhurley · 2025-10-14T10:09:40Z

internal/collector/otel_collector_plugin.go

+		return fmt.Errorf("error while marshaling to YAML: %w", err)
+	}
+
+	writeErr := os.WriteFile("/var/lib/nginx-agent/opentelemetry-collector-agent-debug.yaml", b, filePermission)


can you make "/var/lib/nginx-agent/opentelemetry-collector-agent-debug.yaml" a constant?

Would it be better to write the file to the location of the default YAML file if it is present?

Something like:
newFileName := "opentelemetry-collector-agent-debug.yaml"
folderPath := filepath.Dir(CollectorConfigPathKey)
DebugFilePath := filepath.Join(folderPath, newFileName)
os.WriteFile(DebugFilePath)

I believe the ask was to put it in var/lib/nginx-agent as the user should not edit this file and that is usually where we place things which we do not want to user to modify

dhurley · 2025-10-14T10:13:48Z

internal/collector/settings.go

-	return []string{cfg.Collector.ConfigPath}
+	configFiles := []string{cfg.Collector.ConfigPath}
+	configFiles = slices.Concat(configFiles, cfg.Collector.AdditionalPaths)
+	slog.Info("Additional config files:", "", configFiles)


Suggested change

slog.Info("Additional config files:", "", configFiles)

slog.Info("Merging additional OTel config files", "config_files", configFiles)

dhurley · 2025-10-14T10:15:05Z

internal/config/types.go

-		Pipelines  Pipelines  `yaml:"pipelines"   mapstructure:"pipelines"`
-		Receivers  Receivers  `yaml:"receivers"   mapstructure:"receivers"`
+		ConfigPath      string     `yaml:"config_path"             mapstructure:"config_path"`
+		AdditionalPaths []string   `yaml:"additional_config_paths" mapstructure:"additional_config_paths"`


Suggested change

AdditionalPaths []string `yaml:"additional_config_paths" mapstructure:"additional_config_paths"`

AdditionalConfigPaths []string `yaml:"additional_config_paths" mapstructure:"additional_config_paths"`

codecov · 2025-10-14T11:16:39Z

Codecov Report

❌ Patch coverage is 70.17544% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.92%. Comparing base (383ce95) to head (b781470).

Files with missing lines	Patch %	Lines
internal/collector/otel_collector_plugin.go	48.14%	10 Missing and 4 partials ⚠️
internal/config/types.go	50.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1314      +/-   ##
==========================================
+ Coverage   84.88%   84.92%   +0.04%     
==========================================
  Files         100      100              
  Lines       14520    14566      +46     
==========================================
+ Hits        12325    12370      +45     
+ Misses       1734     1728       -6     
- Partials      461      468       +7

Files with missing lines	Coverage Δ
...llector/nginxossreceiver/internal/config/config.go	`100.00% <ø> (ø)`
...er/internal/scraper/accesslog/nginx_log_scraper.go	`83.13% <100.00%> (ø)`
...internal/scraper/stubstatus/stub_status_scraper.go	`90.83% <100.00%> (ø)`
internal/collector/nginxplusreceiver/config.go	`76.00% <ø> (ø)`
internal/collector/nginxplusreceiver/scraper.go	`73.07% <100.00%> (ø)`
internal/collector/settings.go	`73.33% <100.00%> (+1.24%)`	⬆️
internal/config/config.go	`85.67% <100.00%> (+0.10%)`	⬆️
internal/config/flags.go	`100.00% <ø> (ø)`
internal/config/types.go	`83.33% <50.00%> (-2.09%)`	⬇️
internal/collector/otel_collector_plugin.go	`80.67% <48.14%> (-1.48%)`	⬇️

... and 5 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 383ce95...b781470. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sean-breen · 2025-10-14T14:23:42Z

Can you add unit tests for the new functions including all potential error paths?

aphralG added 3 commits October 1, 2025 14:41

added agent config var

19ac8d1

embedded otel collector

69b27ae

embedded otel collector

4206ac1

aphralG self-assigned this Oct 1, 2025

aphralG requested a review from a team as a code owner October 1, 2025 16:43

github-actions bot added the chore Pull requests for routine tasks label Oct 1, 2025

aphralG changed the title ~~Embedded Otel Collector~~ Embedded OTel Collector Config Oct 1, 2025

fix test

ee804c8

spencerugbo reviewed Oct 6, 2025

View reviewed changes

dhurley reviewed Oct 14, 2025

View reviewed changes

PR feedback

51c5847

aphralG requested review from dhurley and spencerugbo October 14, 2025 11:05

Merge branch 'main' into embedded-otel-collector

a3e0e8f

aphralG added 3 commits October 15, 2025 11:58

unit test

b9f0e8f

unit test

967f8f0

update code coverage settings

b781470

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Embedded OTel Collector Config #1314

Embedded OTel Collector Config #1314

aphralG commented Oct 1, 2025

Uh oh!

spencerugbo Oct 6, 2025

Uh oh!

aphralG Oct 6, 2025

Uh oh!

dhurley Oct 14, 2025

Uh oh!

Akshay2191 Oct 14, 2025 •

edited

Loading

Uh oh!

aphralG Oct 14, 2025

Uh oh!

dhurley Oct 14, 2025

Uh oh!

dhurley Oct 14, 2025

Uh oh!

codecov bot commented Oct 14, 2025 •

edited

Loading

Uh oh!

sean-breen commented Oct 14, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	- /etc/nginx-agent/nginx-agent-otelcol.yaml
	- "/etc/nginx-agent/nginx-agent-otelcol.yaml"

	slog.Info("Additional config files:", "", configFiles)
	slog.Info("Merging additional OTel config files", "config_files", configFiles)

	AdditionalPaths []string `yaml:"additional_config_paths" mapstructure:"additional_config_paths"`
	AdditionalConfigPaths []string `yaml:"additional_config_paths" mapstructure:"additional_config_paths"`

Embedded OTel Collector Config #1314

Are you sure you want to change the base?

Embedded OTel Collector Config #1314

Conversation

aphralG commented Oct 1, 2025

Proposed changes

Checklist

Uh oh!

spencerugbo Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

aphralG Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

dhurley Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Akshay2191 Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aphralG Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

dhurley Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

dhurley Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sean-breen commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Akshay2191 Oct 14, 2025 •

edited

Loading

codecov bot commented Oct 14, 2025 •

edited

Loading

sean-breen commented Oct 14, 2025 •

edited

Loading